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ABSTRACT 



This study develops a computer assisted instruction 
response evaluation technique to aid CAI lesson writers in 
course development. The objectives for this technique was 
to experimentally identify principal incorrect response 
forms and determine characteristic error patterns for a 
given concept to be taught. Using a specific concept, the 
proposed technique was evaluated. Sixty-one subjects were 
randomly selected from three different universities and 
assigned to solve a problem-solving task. Using mental 
imagery and oral reporting, they produced the necessary 
protocol for this experiment. Although the response eval- 
uation technique developed in this study was reasonably 
successful, a specific response error pattern for each 
subject could not be determined. 
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I. 



INTRODUCTION 



Abrams (1971) asserts that the most difficult problems 
impeding the progress of computer-assisted instruction (CAI) 
are those involving the basics of the educational process 
itself and the inherent complexity of CAI from the point of 
view of the educators. While a good CAI system will not 
appear complex to the students and the instructors who use 
it, the existing and proposed new system are difficult for 
even knowledgeable educators to evaluate for applicability 
and desirability. 

Even more basic is the problem of the use of diverse 
forms of responses to indicate the attainment of knowledge 
and understanding. This problem is associated with two 
questions : 

(1) What type of response is required from the 
student and the CAI system for a given 
educational evolution? 

(2) Will this response be a true indicator of 
the student's understanding of the concepts 
involved? 

These are difficult, nevertheless, important questions as 
indicated by Simon: "the possibilities that are opened up 

for improving human problem-solving, thinking, and decision- 
making activities" are "at least as important" as the 
genuine revolutionary "developments [which are now apparent] 
in automation." jweissinger-Baylon, 1978, p. l] . 
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The author shares both Abram's concern and Simon's con- 
fidence for this intrinsic problem area of computer- assisted 
instruction. Although much has been accomplished in this 
problematical area, much remains to be solved. Until the 
many factors involved in the response evaluation process is 
more fully explored and more thoroughly understood, it is 
the author's opinion that it will be difficult for CAI to 
realize its full potential. 

There have been many variations in the definition of 

computer-assisted instruction (CAI) in the past two decades. 

The one definition that encompasses the whole field and 

which will be used in this paper is best defined by Abrams: 

. . . the use of computers to aid teachers and students in 
the educational process, utilizing such functions as pre- 
senting materials and problems, guiding a student's path 
through a course by selecting the material to be pre- 
sented or by assigning tasks to be completed away from the 
computer, or any combination of these functions. 

(Abrams, 1971, p. 4j 

Therefore, CAI does not just simply imply typing out 
questions to which students respond and by then giving cues 
and different hits, CAI includes the computational use of 
the computer, when appropriate, as part of the instruction. 

Computer-assisted instruction systems may be generally 
viewed in two ways: machine-directed and student-directed. 

In the machine-directed approach, various paths and alterna- 
tives through instructional material is programmed into a 
machine. In this approach there must be prior knowledge of 
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the needs of the student and a predetermined method by which 
the computer can evaluate the student's current state of 
knowledge. In addition, the student's past performance 
record and techniques for describing a further path for each 
student, which would optimize his learning experience, must 
be at the machine's disposal. 

In the student-directed approach, the sequence of the 
material presented is altered only at the request of the 
student. The approach uses the aspect of learning by dis- 
covery and experimentation. This area, however, has been 
almost untouched by current research and, therefore, much 
less has been done to utilize this type of approach. 

This paper deals primarily with the machine-directed CAI 
approach. A new computer-assisted instruction (CAI) tech- 
nique with the potential to improve the effectiveness of 
CAI was developed and evaluated. This technique experi- 
mentally identifies the principal incorrect forms of any 
given concept that is taught. Characteristic error patterns 
are associated with incorrect forms so that the subject may 

9 

be presented with a CAI response tailored to that particular 
incorrect concept that he has acquired. New instructional 
material or other techniques could then be utilized depend- 
ing upon the student's understanding or lack of understand- 
ing of specific concepts. 

Absence of any contributing literature in the area of 
CAI response evaluation and personal communications with 
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prominent individuals in the area of CAI, indicate that the 
approach discussed in this study, is the first significant 
attempt to provide a specific set of rules or guidelines 
for evaluating the subject's responses. This does not im- 
ply that the response evaluation area of CAI was never con- 
sidered or discussed, but that a set of rules has never 
been formalized. The following method has therefore been 
designed to provide CAI lesson writers with an additional tool 
for structuring CAI course material. 
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II. BACKGROUND 



A. THE NEED FOR A RESPONSE EVALUATION TECHNIQUE 

The use of various computer-assisted instruction systems 
has been on the upswing in the last decade. This growth has 
been desirable and is a result of the increased amount of 
human knowledge that needs to be attained, the ever increas- 
ing number of students to be taught, and the availability of 
qualified instructors. Early studies revealed that consist- 
ent time savings occurred during a three year experiment 
using microcomputers to manage students' self-paced pro- 
gression through a course (Evans and Johns, 1979). Orlansky 
and String (1979) , in an iconoclastic attack on the cost- 
effectiveness of thirty studies of military computer-based 
education (CBE) have also concluded that a thirty percent 
time savings is possible. It can therefore be generally 
agreed that CAI can achieve educational objectives more effi- 
ciently and more effectively than other teaching methods. 
However, four basic educational factors accentuate the need 
for a computer- as sis ted instruction (CAI) response evalua- 
tion technique . 

The first factor is the trend toward individualized 
instruction. The fewer the number of students for which a 
teacher is responsible, the more time and individual 
attention each student will receive. Therefore, from a 
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theoretical aspect, the more each student will gain from his 
educational experience. Conventional mass media such as 
books, films, television, and even programmed instruction 
are inherently incapable of individualization. Books, films, 
and television programs would have to be rewritten to fit 
each student, and conventional programmed instruction systems 
lack the flexibility and decision-making capability required 
of a truly effective individualized system. CAI seems to 
have the potential to meet these requirements. However, the 
current relatively sophisticated computer-assisted instruc- 
tion systems are only tailored towards students who are 
homogeneous in background, ability, cognitive style, and 
motivation. The systems are incapable of individualization 
in the sense of structuring a response as a true indicator 
of the student's understanding of the concepts involved. 
However, by evaluating the subject response and then pro- 
viding a tailor-made CAI response to reflect understanding 
or lack of understanding of a given concept, seems to have 
the potential to improve the effectiveness of CAI and its 
value to individualized instruction. 

The second factor is the evaluation of the student's 
learning progress. Although, sophisticated CAI systems 
keep track of the number of wrong responses for a particu- 
lar concept, they do not identify to the educator the 
incorrect concept that the student learns. Using a response 
evaluation technique, the wrong concept may be identified 
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and new instructional material, review material, drill and 
practice sessions, tutoring, or other techniques could then 
be utilized depending on the student's under standing. 

The trend towards the use of microcomputers is the third 
factor. Computer-assisted instruction has typically been 
applied using large, dedicated computer systems and special 
CAI languages. There are virtually no limitation on memory 
or file sizes for the instructional material. However, 
microcomputers are now being sold at prices comparable to 
color television sets and their use in educational settings 
will skyrocket in the next few years. Invariably, CAI will 
be one of the uses to which these computer systems are put. 
These systems, however, have inherent limitations. The most 
important being memory size. Since, these limitations are 
fixed attributes of the systems , they must be dealt with 
creatively. One method is to use a response evaluation 
technique. Using such a technique, the size of the programs 
may be reduced by limiting the required number of paths or 
alternatives through the instructional material. 

Finally, the fourth factor is the limited applicability 
of computer-assisted instruction to university level subjects. 
Weissinger-Baylon asserts that according to Atkinson, "the 
relatively sophisticated computer-assisted instruction 
systems that have been successfully introduced for reading 
and elementary areas are neither applicable to under- 
graduate mathematics nor to other university level subjects; 

too little is known of the structure of the required skills 
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or of the psychological processes for their mastery." 
the type of response for certain concepts are more fully 
explored and more thoroughly understood it will be difficult 
for CAI to realize its full potential. 

B . RESEARCH OBJECTIVES 

The most difficult problem impeding the progress of com- 
puter-assisted instruction are those involving the basics of 
the educational process itself. The type of response that 
must be obtained from the student to get an indicator of his 
true understanding of the concept, is a difficult question. 
Although there has been some work performed by Newell, Simon, 
Shaw, and Weissinger-Baylon in the area of human problem 
solving (Weissinger-Baylon, 1978, p. 5] , it has only been 
applied to the area of computer-assisted instruction in a 
very limited amount. Substantial progress in this direction, 
however, will permit far reaching new approaches to education. 

The present study is conceived as one step in a longer 
program, the development of major improvements, efficiency 
and value of computer-assisted instruction systems. The 
immediate objective is to develop and evaluate a new com- 
puter-assisted instruction (CAI) technique using mental 
imagery and verbal responses to experimentally identify the 
principal incorrect forms of a given concept that the stu- 
dents usually learns. Characteristic error patterns will be 
identified and associated with incorrect concept forms on a 
diagnostic test and will be evaluated in regards to its 
application to a computer- as sis ted instruction system. 
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C. TASK SELECTION 



Task selection for the development and evaluation of any 
new technique or procedure must be based on many factors and 
therefore makes it a difficult chore. Weissinger-Baylon , 
from his research, states that the most important factors in 
task selection are: "emphasis on tractability or support 

for the experimenters hypothesis must be minimized; the re- 
search must have application, at least potentially; and 
finally, the complexity of the task must not limit the 
communicability of the findings." The author agrees with 
Weissinger-Baylon' s findings, but adds that a task may be 
selected because of it's interest to the analyst and because 
it complements the work of others . 

The choice for the selection of mathematics as a field 
to test and evaluate responses for a CAI system was made 
as a result of its many advantages: "for the chosen problem 

area, both concept and vocabulary are highly developed; ade- 
quate intellectual resources are available for this topic; 
its method and administrative procedures are convenient; 
and it is an important area of application, controlling al- 
most all entries to attractive professions." 

Determining a problem area within the chosen field was 
the next stage in the selection process. It was a difficult 
task. The material had to satisfy both the requirements of 
meeting the objectives of this research and, most impor- 
tantly, be within the grasp of existing information 
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processing methodologies. 

Weissinger-Baylon* s personal communication with Karl 
deLeeuw [Weissinger-Baylon, 1978, p. 3$ , proposed an 
analysis of a group of closely related concepts having to 
do with "bounding" of sets. It was felt such a concept 
would adequately propose or illustrate, at least theoreti- 
cally, the technique to evaluate and analyze responses for 
a CAI system. The problem that arose, however, was what 
type of bounding concept to use — the greatest lower bound, 
the least upper bound, or determine if the set is bounded 
above or below. 

Preliminary studies conducted by Weissinger-Baylon on 
the proposed concepts , suggested that the focus of the 
analysis for a response evaluation process for a CAI 
system, be narrowed to a single definition. His studies 
on the upper bound concept produced the most interesting 
results and it was determined that the importance of the 
upper bound concept, its manageability, and Weissinger- 
Baylon 's prior research in this area, made this concept 
the most attractive candidate for this research study in 
evaluating the response process for a CAI system. 

D. METHODOLOGICAL OVERVIEW 

The development of a suitable methodology proceeded in 
stages. Test results from a research study conducted by 
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Weissinger-Baylon on the upper-bound of sets concept, were 
made available for this research, and pre-empted the need 
for duplicate and redundant testing in the initial phase of 
the study. 

1 . Subjects 

The sixty-one students who served as subjects were 
volunteers from three different universities: the Naval 

Postgraduate School, Stanford University and Colorado State 
University. Subjects were selected to provide a reasonable 
balanced representation by sex, specialization and profes- 
sional levels. They were obtained from bulletin board 
announcements, distribution of sign-up sheets and by 
personal request. With the exception of the Stanford Uni- 
versity students, who received a $6.00 fee, no other 
compensation was offered for the subject’s participation in 
this experiment. The mix consisted of 23% females and 77% 
males. Of these, 33% were graduate or undergraduate stu- 
dents in the mathematical sciences (Computer science. 
Statistics, Operations research and Mathematical sciences) 
and 67% were undergraduates or non-mathematical science 
graduates. Prior research had dictated that subjects with- 
out university mathematics would experience difficulty with 
this particular concept and were therefore required to have 
had at least two years of calculus. 
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2 . Design 

The problem-solving task was initiated by looking 
at fifteen different, but related definitions from advanced 
calculus. From the results, a final version was produced, 
which focused on one definition: the upper bound of a set 
of real numbers. The criteria for an upper bound of a set 
was met if there existed a real number which was greater 
than every element of the set. 

Nine problems were chosen from standard texts of ad- 
vanced calculus and grouped into the following four types 
of sets: 



( 1 ) 

( 2 ) 

(3) 

(4) 



In addition, 
methods : 



finite sequences: e.g. (1,2, 3, 4} 

intervals: e.g. (a,b) 

convergent infinite sequences: 
e.g. {l, 1/2, 1/3, ...1/n,...} 

divergent infinite sequences: 
e . g . { 1 , 2 , 3 , . . . n , . . . J 

(Weissinger-Baylon, 1978, p. 3 2] 

the above sets were presented in the following 



(1) enumerative: {1,2, 3, 4} 

(2) nominal: N, the natural numbers 

(3) attributive: finite set 

[Weissinger-Baylon, 1978, p. 32] 
The sets were randomized by type and presentation method. 

These representations vary slightly from the final de- 
sign since they were modified to make them suitable for a 
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wide range of subjects. Further, it is well to note that 
the degree of accuracy will be based on the degree to which 
the subject's responses match the existing textbooks. The 
production of imagery or step by step analogy of the prob- 
lem set was not consciously a criterion. 

3 . Procedure 

The detailed, step-by-step procedure for the CAI 
response evaluation technique is given in chapter 3 . The 
procedural model, as designed, consists of seven major 
components: selection of target content, design of 

problem-solving task, selection of protocol methodology, 
pretesting, refining the task, administering the final 
version of the task, and analysis of results. 

The procedural methodology that was used for adminis- 
tering the problem-solving task consisted of contacting 
the subject prior to the experiment and agreeing on a set 
time for administration of the task. Before beginning the 
problem-solving task, the subjects were interviewed to 
determine their qualifications for testing on the 
particular concept. In addition, during the interview the 
subjects provided the following information: year. 
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department, and educational and career plans. When the 
participant was found to be qualified, he was given the 
following instructions: 

Instructions . On the next page, you will find a defini- 
tion taken from an undergraduate mathematics course. At 
first the language may seem imposing, but the concept is 
not difficult. Please read carefully, and try to under- 
stand it as well as you can. Next, go on to the practice 
problem and the questions which follow: 

We would like you to think out loud as much as possible. 

If you use any mental imagery in your thinking, please 
draw or describe it. At the end of each problem, you 
are to write a complete summary of your thought process. 

You may begin when you wish. There is no time limit. 
Please ask any questions which you feel are necessary. 

Do not forget to think out loud, or draw or describe any 
mental imagery you may have . 

(Weissinger-Baylon , 1978, p. 36j 
A tape recorder was then put in front of the subject to 
report his oral responses, if any occurred. When the 
subject indicated that he understood the instructions, he 
was given the test booklet and then asked to start the 
exam. The examiner was seated beside the subject to 
observe and take notes. At the end of the entire session, 
the subject was again interviewed to clarify and explain 
the exam and, in some cases, uncover errors in the written 
protocol without the risk of biasing the responses. 

In summary, the administration process was designed to 
systematically obtain data using the following protocol 
methodology : 

1. Verbal protocol — to clarify infrequent ambiguities 
in the written summaries. 



22 



2 . 



Graphic protocol — to illustrate a path of the 
subject's thought processes. 

3. Written summaries — to obtain a formalized/ step-by 
step analogy of the problem. 

4. Interaction with experimenter — to obtain additional 
clarification when imagery explanations were 
ambiguous and when steps were omitted during the 
written summaries. 

The importance of this data collection method is demon- 
strated in the data analysis phase of the procedural model. 
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III. DEVELOPMENT OF A RESPONSE EVALUATION TECHNIQUE 



A response is a unit of behavior and the building block 
of complex performances. The flick of an eye and the twitch 
of a finger are examples of simple responses; eating, walk- 
ing, speaking and reading are all instances of more complex 
responses. A primary objective of educational technology 
is the guidance of individual's responses. To accomplish 
this objective, the educator must first define and enumer- 
ate the components of the performance, that is, the 
responses, that he wishes to produce. It is then possible 
to arrange the stimulus conditions which will result in the 
desired response. It also becomes possible to develop 
objective measures of the frequency and accuracy of the 
response . 

The consequences of a student's response are extremely 
important in learning. The events which follow the occur- 
rence of a response have an effect upon future behavior. 
Examples of such response consequences are reward, punish- 
ment, and knowledge of results. The reward or other stimu- 
lation provided to the learner immediately after responding 
is a significant factor which seems to determine whether 
learning takes place. The occurrence of certain consequen- 
ces of behavior that are effective in producing and 
maintaining behavior is called reinforcement by behavioral 
psychologists. In an educational situation, the instructor 
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controls the consequences of the student's behavior. Since 
these consequences determine whether the student learns , 
the educator will want to maximize those consequences which 
facilitate learning. 

Definition of an effective and efficient procedural 
model for a response evaluation methodology began with an 
analysis of the characteristics of the instructional environ- 
ment. This examined specifically the environment defined 
by the evaluation of course mastery, and the types of in- 
structional problems for which a response evaluation process 
could provide solutions in this environment. 

The procedural model, as derived, consists of seven 
major components, each with a number of constituent steps. 

The components are selection of target content, design of 
problem solving task, selecting protocol methodology, pre- 
testing, refining the task, administering the final version 
of the task, and analysis of results. The full model is 
shown in figure 1. The model's seven component steps are 
described in the following paragraphs. 

A. SELECTION OF TARGET CONTENT 

The first component of the procedural model (see figure 
2) assumes the presence of a main track of non-CAI materials 
and defines criteria for selecting those materials for which 
alternative CAI modules are to be developed. The steps in 
this component do not constitute a precise algorithm for 
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FIGURE 1. This flowchart summarizes the main steps in the 
experimental design and administration of a 
response evaluation procedure. 
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FIGURE 2. Selection of Target Content 



content selection. Rather the intent is to identify a 
number of factors which should be considered in determining 
target content selection. Few hard and fast rules can be 
defined, and value judgements play a major role in the 
decision process. 

The first step in the selection of target content is to 
select a course or courses where the use of CAI will be 
beneficial to the training process. This should be a 
school-level decision based on considerations as training 
cost per student, instructor student ratios, and types of 
training materials used. 

The principal consideration must be to choose a course 
where CAI materials can improve motivation, and where the 
unique capabilities of CAI will provide needed information. 

After selecting a course, the next step is to select a 
specific concept. Here, the selection considerations are 
the importance of the particular concept, concept failure 
rates, variability of concept mastery test scores, and aver- 
age times and variability of times to complete the concept. 

When the two selection steps (course and concept) are 
completed, it must be determined if CAI is feasible for the 
tentative selections. This is where preliminary knowledge 
of the system plays an important role. Possible reasons for 
determining non-feasibility of CAI include necessity for 
graphics or simulations which are beyond the capabilities of 
the available system and personnel, or too few concept 
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selections to justify hardware or development cost. 

With the final determination of CAI feasibility, the 
educator is now able to define and enumerate the components 
of performance, in other words, the response he wishes to 
produce for a given task. The educator is ready to begin 
the next component of the procedural model, the design of 
the problem-solving task. 

B. DESIGN OF PROBLEM SOLVING TASK 

The steps of the design and the development of a 
problem-solving task component is shown in figure 3. It 
assumes that the course and concept selection together with 
their CAI feasibility have already been determined. 

In starting to develop a new teaching session, an edu- 
cator must be given a certain amount of basic preliminary 
information about the material and system to be used. This 
may be provided in the form of text, drawings, diagrams, 
moving pictures, or any combination of these modes. This 
process will ensure that the finished materials avoid such 
problems as omitting necessary content, covering objectives 
incompletely or at an incorrect level, incorrect or inade- 
quate emphasis on particular subjects or concepts, incom- 
plete material supplements, poorly designed teaching 
strategies, and tests which do not address the objectives. 
Once the workings of the system and the basics of the 
materials are understood, development of the task to obtain 
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FIGURE 3. Design of Problem Solving Task 
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proper responses from the user and the system may begin. 

After this initial phase, the design of the problem- 
solving task should begin with a careful analysis of the 
instructional requirements. The forming of the instrument 
to evaluate a particular concept is obtained through the 
educational process of an instructors preliminary knowledge 
of the subject matter. In this phase, it is important that 
the test has face validity for the objectives which addres- 
ses the particular concept. It is also important that the 
problem-solving task cover the specific concept matter at 
the appropriate difficulty levels. Since this instrument is 
the principal element which will guide the response evalua- 
tion process , it must fully represent the intent of the 
educator to assure adequate performance. Any revisions must 
be reviewed by the educator to insure the technical accuracy 
of the content changes . 

Unless these first two steps are successfully completed, 
all subsequent efforts may be jeopardized. For example, if 
the problem-solving task is so designed that it will not 
evaluate the particular concept, the work in the evaluation 
of principal incorrect responses for that particular concept 
will be largely wasted. 

The final step of the design of the problem-solving task 
includes the determination of the proper responses desired 
and the possible major incorrect responses that may be 
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obtained. Since, the number of possible incorrect responses 
for any particular concept is theoretically infinite, this 
process will only serve as an initial starting point in the 
identification of the principal incorrect forms of that con- 
cept that is taught. 

C. SELECTION OF PROTOCOL METHODOLOGY 

The protocol methodology for the administering and eval- 
uation of the problem-solving task (see figure 4) are given 
as a set of rules. Revision of the protocol may be made at 
any time it is determined that it is non-feasible for a 
particular application. The rules are as follows: 

1. Selection of students for use on the problem-solving task 
should be on a volunteer basis. Using volunteers rather than 
individuals ordered to participate in the problem-solving 
task will avoid some possible attitude problems. 

2. Volunteers should be selected on their qualification for 
the particular concept being tested. They should be se- 
lected to provide a reasonably balanced representation by 
sex, specialization and professional level. 

3 . The problem-solving task should be conducted outside the 
classroom and should be informal. Attempting to work within 
the classroom is likely to disrupt regular classroom activi - 
ties. In addition, it has been found that the problem- 
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FIGURE 4. Selection of Protocol Methodology 
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solving task should be administered to one test subject at 
a time. Group testing tended to inhibit the subject's will 
ingness to verbally report their thought processes. 

4. The examiner should be available to answer questions, 
provide guidance where necessary, and note areas where 
students have difficulty. 

5. Finally, and most important, the test subjects are to 
learn the particular concept and apply it to a number of 
practice problems. During the problem-solving task, the 
test subjects are to verbally report their thought proc- 
esses and draw and describe mental images, if any occurred. 
In this way, the subject's thought processes for a partic- 
ular concept may be determined by analyzing the result and 
the sequence of steps that were used to obtain that result. 

The above rules provide effective guidelines for an 
efficient and standard administration and evaluation of 
the problem-solving task. It is noted that verbal re- 
ports of a subject's thought processes may be tape re- 
corded for accuracy and historical references. This is 
only required, however, if it has been found that test 
subjects tend to lean towards verbal reporting vice the 
written method. Continual evaluation of the methodology 
must be made during the problem-solving task for any 
particular situation. 
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D. PRETEST 



The fourth component of the procedural model (see 
figure 5) assumes that a dedicated amount of time has al- 
ready been spent in the selection and development of a 
problem-solving test. The purpose of this phase is not so 
much directed at evaluating the response effectiveness of 
the problem-solving task as it is at locating errors of 
content and logic which were overlooked. Typically, the 
number of students tested would not be sufficient to answer 
questions concerning the incorrect concepts that the student 
learns, but do provide an avenue by which the educator can 
learn if the materials are easily understood by their in- 
tended audience. The pretest provide answers to questions 
such as the following. Are there inconsistencies between 
the concept and the problem-solving task? Are directions to 
the student clear enough to be followed? Is wording clear? 

Is the problem-solving task too long to hold the student's 
attention? 

The pretest should generally be administered to 12-15 
students, but the number will vary with the number of errors 
found early in the pretest. If many errors are found with, 
for example, the first four or five students, it would be 
preferrable to suspend the pretest until the problems have 
been resolved. If available manpower permits, it may be de- 
sirable to supplement review of the pretest by other instruc- 
tional personnel for the purpose of locating grammatical and 
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FIGURE 5 




. Pretest 
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typographical errors and problems with clarity. 

The next step pertains to data analysis. With the use 
of imagery, as specified in step 3 of the procedural model, 
principal incorrect forms of the concept that a student 
learns may now be initially determined. These incorrect 
concept forms are to be compared to the theoretical incorrect 
forms established in step 2 of the procedural model. After 
comparison of both the theoretical and diagnosed forms of the 
principal incorrect responses , the educator is now ready to 
begin the next component of the procedural model, modifica- 
tion of the problem solving task. 

E. MODIFICATION OF THE PROBLEM SOLVING TASK 

This component (see figure 6) of the procedural model is 
designed to modify or correct any errors and ambiguities re- 
vealed by the pretest. It*s purpose is to evaluate the 
instructional effectiveness of the materials to determine 
the specific areas of weakness. Problems which have more 
than one theoretical incorrect response or too many diagnosed 
responses, such that a specific incorrect concept cannot be 
applied to the problem, are to be evaluated and deleted from 
the problem-solving task. If it is found that the problem- 
solving task has too many unresolvable problems, it must be 
determined if the particular subject matter is actually 
testable. If not, the procedural process for a response 
evaluation procedure terminates at this point, otherwise the 

educator is ready to begin the next component of the 
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FIGURE 6. Modification of Problem-solving Task 



procedural model, administration of the problem-solving task 
for formative evaluation. 

F. ADMINISTERING OF THE PROBLEM SOLVING TASK 

The sixth component of the procedural model (see figure 
7) assumes that the problem-solving task has now been per- 
fected and is ready to be used to evaluate the incorrect 
responses for the particular concept. A reasonably balanced 
population of test subjects by sex, specialization and pro- 
fessional levels are to be provided. The test is to be 
administered in the same methodology as was used in the pre- 
testing of the instrument. 

The final study should be administered to 20-30 students, 
but the number will vary depending on the particular con- 
cept that is being evaluated. Successful completion of this 
component moves the sequence of the response evaluation 
process to the last component of the procedural model, data 
analysis . 

G. DATA ANALYSIS 

The analysis of the results appears in the final chap- 
ters. However, a brief outline may be useful in interpret- 
ing the data analysis (see figure 8) . 

The first step examines the incorrect form of the 
responses by subject and by problem. The influence of abil- 
ity level, sex and type of problem are considered next. Final- 
ly, the data will be analyzed as its applicability to CAI. 



39 



FIGURE 7 




Administering Problem-Solving Task 
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FIGURE 8. Data Analysis 
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IV. DATA ANALYSIS 



The final study included sixty-one subjects. Thirty- 
seven of these were obtained from prior research 
(Weissinger-Baylon, 1978) and twenty-four (see Appendix A) 
were obtained during the course of this study. They provide 
a group reasonably balanced by sex, specialization, profes- 
sional level,and most important, ability to understand the 
particular concept — the upper bound of a set. 

The test group was comprised of 23% females and 77% 
males. Of these, 33% were graduate or undergraduate stu- 
dents in the mathematical sciences and 67% were under- 
graduates or non-mathematical science graduates . All 
subjects were volunteers, and the time required to do this 
experiment varied from 10-29 minutes per subject. The final 
test instrument consisted of a definition, a practice prob- 
lem, and nine problem-solving sets (see figure 9) . 

As is true in the analysis and evaluation of any problem- 
solving task, the principal challenge lies in summarizing 
the protocol data. Using data from 671 problems obtained 
during this research, the response evaluation technique for 
the upper bound of a set concept was evaluated from three 
different standpoints: (a) evaluation of its ability to 

isolate the major incorrect responses for a particular 
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FINAL TASK DESCRIPTION 



Set Number 


Set 


Definition 


Let E be a subset of the real numbers, R. 
An element, m, of R is said to be an upper 
bound of E if x is equal to or less than m 
for every x in E 


Practice 

Problem 


Let E be the set £l,2, 3, 4^, i.e. the set 
consisting of the first four positive 
integers. Does E have an upper bound? 


Set 2 


The set of natural numbers, N (i.e., the 
positive integers) . 


Set 4 


A finite set. 


Set 5 


(l, 1/2, 1/3, ...} 


Set 6 


The set of real numbers, R. 


Set 11 


The non-negative numbers 


Set 13 


The closed interval [3,5] 


Set 17 


{ , /3 , /3T , , rz , , ...} 


Set 18 


{tan x, where x =tf, 2 rf, 3 Tf, 4 rf, 5 rf, 6rf, ...} 


Set 21 


{l, 2, 4, 8, 16, 



Figure 9. The sequence of steps to perform the task consisted of: 

(1) reading the definition, (2) and then solving the practice problem* 
Each subject was then instructed to determine whether or not each of 
the sets had an upper bound. During the process of the problem solving 
task, subjects were asked to provide verbal and written reports of 
their thought processes, and draw mental pictures, if any occurred. 
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concept, (b) evaluation of response error statistics by 
type of set, and type of set between ability level and sex 
and, (c) evaluation of subjects response pattern. Results 
of each of these three evaluation efforts are reported in 
the following subsections. 

A. ISOLATION OF MAJOR INCORRECT RESPONSES 

Prior research has dictated the existence of an in- 
finite number of theoretical correct and incorrect re- 
sponses for any particular concept , depending on the 
nature of the behavior and the level of skill of the indi- 
vidual. [Taber, Glaser, Schaeffer, 1965, p. 19] . Keeping 
this fact in mind, the isolation of major incorrect 
responses becomes of the utmost importance during the 
final evaluation of any problem-solving task. 

From the educators preliminary knowledge of the con- 
cept material and the initial analysis of the pretest 
results — procedural model, step two and four respectively, 
seven major incorrect responses or incorrect concept forms 
were identified as being possible for the upper bound 
concept. They were classified as follows: 

1. Positive infinity is the upper bound for all 
sets . 

2. Infinite number of points has no upper bound. 

3. An infinite set has no upper bound. 

4. The upper bound has to be a member of the set, 
if the set is to have an upper bound. 
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5. Misreading of the problem. 

6 . Misunderstanding the problem. 

7. Other (Ignoring the problem, unable to do 
problem, etc.) 

Analysis of the above incorrect response concepts 
indicated that incorrect concept forms two and three 
(infinite number of points has no upper bound and infinite 
set has no upper bound, respectively) were very similar in 
construct and may be combined to state that: "no upper 

bound exists for an infinite set." In addition, in- 
correct concept form six (misunderstanding the problem) 
may be placed in the incorrect concept form seven (other) 
category, since the relationship between the two concept 
forms were similar in nature. This initial detailed 
analysis of all the possible major concept forms resulted 
in the isolation of five incorrect concept forms for the 
particular upper bound concept. 

Application of the five incorrect concept forms to the 
problem-solving sets, illustrated, theoretically, the 
errors that an individual would or should obtain with each 
particular incorrect concept form that he had learned (see 
figure 10) . It must be noted that the final test instru- 
ment was designed and modified (procedural model, step 
five) so that no more than one theoretical incorrect 
concept applied to each problem-solving set. 
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THEORETICAL INCORRECT CONCEPT APPLICATION SETS (CON'T) 
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incorrect concept is applied 



Using the above information, the final test results were 
analyzed and the errors were evaluated in regards to the 
five major incorrect concept forms (see figure 11) . The 
analysis indicated that 94 percent of the errors fell 
into the particular error category that was determined, 
from the theoretical application of incorrect concept 
forms , to produce an error for the individual problem- 
solving set. In addition, 80 percent of the errors fell 
into just two major incorrect concept categories: plus 

infinity is the upper bound of everything and an in- 
finite set has no upper bound. 

It is reassuring to note that the absence of response 
errors for the major incorrect concept form "the upper 
bound must be an element of the set" indicated that the 
total number of incorrect response forms for this par- 
ticular concept may be further reduced to only four 
major categories. This is a reduction of 43 percent from 
the number initially determined in the early stages of the 
response evaluation model. 

B. SUBJECT AND SET TYPE EVALUATION 

Results of the "subject to set evaluation" is discussed 
in two parts: evaluation of type of sets, and evaluation 

of subject, ability level and sex between type of set. 

The errors for the particular sets were tabulated in 
figure 12. The sets are classified as being either 
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SUBJECT ERRORS BY INCORRECT CONCEPTS 
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ect (Weissinger-Baylon , 1978, pp. 1-224 Appendix), while the "S 
test results in this research project. Appendix A. 



SUBJECT ERRORS BY INCORRECT CONCEPTS (CON'T) 
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SUBJECT ERRORS BY INCORRECT CONCEPTS (CON'T) 
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SET DESCRIPTION, CATEGORIES AND ERRORS 
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ture 12 The set description where presented exactly as above. Each set is classified 
being concrete or abstract. Number and percentage of errors are for 61 test subjects. 



"Abstract," those that may require interpretation, for 
example: "The non-negative numbers," and "Concrete," those 

which subjects can analyze directly, for example: 
"{1,2,4,8,16,...}." Statistical analysis indicated that the 
error rates for the abstract sets were considerably higher 
than the concrete sets (72% and 28%, respectively). The 
difference may be attributed to three significant factors : 
the final instrument consisted of five abstract sets com- 
pared to only four concrete sets; the error rate for set 
eighteen is abnormally high; and due to the inherent 
nature of the abstract type of set. 

Deletion of set eighteen from the test instrument to 
remove the problem solving set with the abnormally high 
error rate and to provide an equal distribution of abstract 
and concrete sets, still resulted in an error rate differ- 
ence between the two types of sets of 60 percent for 
abstract sets and 40 percent for concrete sets. Therefore, 
is the inherent nature of the type of sets a significant 
factor on the type of response? Consider for example set 
eleven : 

Set 11. The non-negative numbers. 

Ten subjects answered it incorrectly: six by incorrectly 

applying the upper bound definition, and four by misunder- 
standing the term "non-negative." Now, consider a mathe- 
matically similar task, set twenty-one: 
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Set 21. {1,2,4,8,16, ...} . 

Both sets are infinite with the only difference that set 
eleven is written in abstract terms while set twenty-one 
is written in concrete terms consisting of five elements. 
Since the differences between the sets are minor, error 
rate comparison is surprising. Indicating that the in- 
herent nature of a problem-solving set is a significant 
factor in the type of response that may be obtained. 

Next, error response for subject, ability level, sex 
and type of set (see figure 13) is analyzed and provided 
below : 

1. It was found that the overall response error rate for 
the problem-solving instrument was 11.7 percent. Even 
though the test was designed to be easy and administered 
to subjects who had the ability to understand the particu- 
lar concept, this indicates that individuals, as problem 
solvers, are often unreliable for objectively easy tasks. 

2. There was a significant difference of 3.3 percent re- 
sponse error rate between the "mathematical science" group 
and the "other" group. The "mathematical science" group 
consisted of graduates from the Naval Postgraduate School 
and Stanford University and undergraduates from Colorado 
State University in the mathematical sciences (Mathematics, 
Computer Science, Statistics, and Operations Research). 
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SUBJECT IDENTIFIERS, SEX, FIELD, LEVEL, AND ERRORS 



Subject 

Number 


Sex 


Field 


MATH 

SCIENCE 


YEAR 


Errors 
No. % 


SI C.D. 


M 


CSM 




GRAD 


1 


11 


S2 L.M. 


F 


cs 


X 


GRAD 


2 


22 


S3 T.V. 


M 


CSM 




GRAD 


1 


11 


S4 R.W. 


M 


CSM 




GRAD 


2 


22 


S5 S.L. 


M 


CSM 




GRAD 


0 


0 


S6 p.H. 


M 


CS 




GRAD 


3 


33 


S7 Y.K. 


M 


CSM 


X 


GRAD 


4 


44 


S8 R.F. 


M 


D.P. 




UND 


1 


11 


S9 F.K. 


M 


D.P. 




UND 


4 


44 


S10 D.M. 


F 


MATH 


X 


UND 


1 


11 


Sll S.W. 


M 


M.E. 




UND 


1 


11 


S12 B.N. 


M 


C .E • 




UND 


3 


33 


S13 E.C. 


F 


MATH 


X 


UND 


0 


0 


S14 G.J. 


M 


C 3 




GRAD 


0 


0 


S15 R.C. 


M 


D.P. 




UND 


1 


11 


S16 K.G. 


M 


D.P 




UND 


1 


11 


S17 E.T. 


M 


D.P. 


X 


UND 


2 


22 


S18 J.L. 


M 


C.S. 




GRAD 


0 


0 


S19 L.N. 


F 


C.E. 




UND 


1 


11 


S20 T.M. 


M 


D.P. 


X 


UND 


4 


44 


S21 A.C. 


M 


D.P. 




GRAD 


1 


11 


S22 J.T. 


M 


C.S. 




UND 


2 


22 


S23 J.B. 


M 


M.E. 




UND 


0 


0 


S24 S.M. 


M 


M.E. 




UND 


2 


22 


T15 R.E. 


M 


O.R. 


X 


GRAD 


0 


0 


T16 J.M. 


M 


C.S. 


X 


GRAD 


5 


55 


T17 P.M. 


M 


M.D . 




M.D. 


3 


33 


T18 L.W. 


XL 


MATH 


X 


Ph.D. 


0 


0 


T19 N.B. 


M 


MATH 


X 


GRAD 


1 


1 


T20 R.P . 


M 


C.S. 


X 


GRAD 


0 


0 


T21 R.R. 


M 


MATH 


X 


GRAD 


0 


0 


T22 J.F. 


M 


M.D. 


X 


GRAD 


0 


0 


T23 D.D. 


M 


STAT 


X 


GRAD 


0 


0 


T24 A.K. 


M 


MATH 


X 


GRAD 


1 


11 



Figure 13. The error values are for nine sets. 
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SUBJECT IDENTIFIERS, SEX, FIELD, LEVEL, AND ERRORS 



Subject 

Number 


Sex 


Field 


Math 

Science 


Year 


Errors 
No. % 


T25 H.S. 


M 


MATH 


X 


GRAD 


0 


0 


T26 S.K. 


M 


MATH 


X 


GRAD 


0 


0 


T27 B.B. 


M 


BIOL 




UND 


0 


0 


T28 J.A. 


F 


Phys/CS 




GRAD 


2 


22 


T29 S.L. 


F 


Lang 




GRAD 


1 


1 


T30 P.W. 


M 


MATH 


X 


Ph.D. 


2 


22 


T31 J.H. 


M 


MATH 


X 


GRAD 


0 


0 


T32 A.S. 


M 


STAT 


X 


GRAD 


0 


0 


T33 G.K. 


M 


STAT 


X 


GRAD 


1 


11 


T34 R.G. 


M 


PSY 




UND 


0 


0 


T35 S.N. 


M 


MATH 


X 


GRAD. 


0 


0 


T36 J.N. 


M 


C.E. 




GRAD 


3 


33 


T37 j.k. 


M 


eng 




AB 


1 


11 


T38 R.B. 


M 


M.E. 




UND 


0 


0 


T39 W.P. 


M 


M.E . 




UND 


0 


0 


T40 D.A. 


M 


PSY 




UND 


1 


1 


T41 N.A. 


F 


EDUC 




GRAD 


- 


- 


T42 J.F . 


F 


C.E. 




UND 


0 


0 


T43 B.B. 


M 


UNDECL 




GRAD 


0 


0 


T44 B.E. 


M 


PHYS 




UND 


4 


44 


T45 J.R. 


F 


UNDECL 




UND 


0 


0 


T46 N.Z. 


F 


UNDECL 




UND 


1 


11 


T47 V.L. 


F 


UNDECL 




UND 


0 


0 


T48 L.H. 


F 


UNDECL 




UND 


1 


11 


T49 C.H. 


F 


MATH 




UND 


0 


0 


T50 J.H. 


M 


C.E. 




UND 


0 


0 


T51 K.T. 


F 


PHYS 




GRAD 


0 


0 


T52 J.G. 


F 


UNDECL 




UND 


0 


0 



Figure 13 (con't) 



The error values are for nine sets. 
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The "other" group consisted of Naval Postgraduate School 
and Stanford University graduates and Colorado State 
University undergraduates in curriculums other than the 
mathematical sciences. The response error rates for the 
groups were: 9.4 percent for the "mathematical sciences," 

and 12.7 percent for the "other" group. This significant 
response error difference provided a strong indication that 
additional years of specialized training does provide an 
improved response performance for technically inclined 
concepts . 

3. The response error rate for males was 12.5 percent, 
for females 8.7 percent. The lower response error rate 
for females is very surprising, since today's society is 
still geared towards the higher technical training of the 
male counterpart. The error rate difference, however, may 
be explained by the unequal test population of females 
compared to males. 

4. As indicated in the set type evaluation, the error re- 
sponse rate for the "Abstract" set category was found to 
be considerably higher than the "Concrete" set category — 
15.1 percent and 7.4 percent, respectively. This is a 
result of the inherent nature of the problem. It has been 
found that individuals relate better to problems which may 
immediately be visualized. Thereby, reducing the amount 
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of time required for the thought process and thus, minimize 
the chance for error. 

5. Response error rates for subject's ability level between 
set type and subject's sex between set type is given in 
figure 14 and figure 15 respectively. The figures indicate 
that mathematical science students are better prepared for 
technical concepts provided in concrete terms, however, 
fail to provide a better score for abstract type problems. 
Moreover, the female is apt to provide better responses than 
their male counterpart for the specific set type categories. 

The above data analysis for the different categories 
indicates that a wide disparity exists between and within 
the groups that were looked at. A split halves reliability 
coefficient, to determine if the nine sets constitute an 
acceptable instrument, was obtained by regressing total 
errors of even numbered subjects on the total for odd 
numbered. The coefficient was r = .66, suggesting that 
the problem-solving task was an acceptable instrument. 

C. SUBJECTS RESPONSE PATTERN 

Analysis of figures ten and eleven, indicated that a 
definite mastery of an incorrect concept existed for some 
test subjects, while for others it was not so easy to 
determine an error pattern (see figure 16) . 
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Figure 15 Response error rates between sex and set type 
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SUBJECT RESPONSE ERROR PATTERN 



S INCORRECT CONCEPT FORM 



u 

B 

E 


Plus 

upper 

sets 


infinity is the 
bound for all 




An infinite 
upper bound 


set has 


no 


The upper bound must 
be an element of the 
set 


C 


No. 




No. 




No. 


No. 




No. No. 


T 


Theor 


• 


Sub j . 


% 


Theor. 


Subj . 


% 


Theor. Subj. % 




Errors 


Errors 
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Errors 


E 


rrors Errors 


















1 


S2 










3 


1 


33 




S3 










3 


1 


33 




S6 


5 




3 


60 










S7 


5 




3 


60 


3 


1 


33 




S8 










3 


1 


33 




S9 


5 




4 


80 










S12 


5 




3 


60 










S17 










3 


1 


33 




S20 










3 


3 


100 
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3 


1 


33 




S22 










3 


2 


66 




S24 










3 


2 


66 




T16 


5 




4 


80 










T17 










3 


2 


66 
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3 


2 


66 




T30 


5 




2 


40 










T33 










3 


1 


33 




T36 


5 




3 


60 










T37 










3 


1 


33 




T40 










3 


1 


33 




T44 


5 




4 


80 










T48 










3 


1 


33 












Total 


40 




26 




45 


21 




0 0 


Avg 


3 


.25 


65% 




1.35 




47% 


0 0% 



Figure 16 . The number of subject errors are compared to the number 
of problem sets that should have theoretically produced error with 
the particular incorrect concept form. 
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Consider, for example, the major incorrect concept 
form "plus infinity is the upper bound for all sets." 
Subjects S9, T16 and T44 responded incorrectly, using the 
particular incorrect concept, to four out of five problem' 
solving sets which were theoretically designed to produce 
errors if that particular incorrect concept form was 
applied. Their test results definitely indicated a mas- 
tery of the incorrect concept form as illustrated in the 
following S9 , T16 and T44 excerpts for set eleven: 



S9 : "Since plus infinity is the largest possible 

number, it must be the upper bound." 

T16: "Bounded by continuum i.e., infinity of con- 
tinuum greater than infinity of natural 
numbers . " 

T44: "Then I remember very strongly that infinity 
is the largest. So u.b. is infinity." 



The other missed sets for the particular incorrect re- 
sponse concept produced a similar type of incorrect reason- 
ing process from the three test subjects. 

Now consider subject T30's response for the same 
incorrect concept form. T30 responded incorrectly to only 
two out of five problem sets. Excerpts from the missed 
problem sets are illustrated below: 

Set 2: "Nothing can be to right of all of them, 

therefore, upper bound exists." 

Set 17: "Square root is unbounded (i.e., off to 

right). Therefore, upper bound exists." 



61 



The response was implicit, but not stated as clearly. 
Subject T30 seemed to follow the correct process for an 
infinite set being unbounded, but then stated "therefore, 
upper bound exists." 

The analysis indicated that the other errors produced 
by the test subjects followed a similar pattern as was 
shown above. With the exception of S20, no other test 
subject answered all of the problem sets, designed to pro- 
duce an error for a particular concept, incorrectly. 

It is surprising to note that an error response 
pattern for each subject could not be determined. There- 
fore, is it meaningful to even talk of error patterns? 
Brunner, Goodnow, and Austin (1956) discussed the role of 
errors and indicated that a notable flexibility of the 
subject's behavioral pattern exists. Different 

strategies will be formed for certain information. 

♦ 

Further explanation for inconsistencies in error response 

patterns is provided in the following statement: 

. . . heuristics do not guarantee any solution at 
all; all that can be said for a useful heuristic 
is that it offers solutions which are good enough 
most of the time. 

(Feigenbaum and Feldman, 1963, p. 6] 

This description is consistent with the behavior of 
the subjects error patterns. 

The error pattern for test subjects with the in- 
correct concept form "the upper bound must be an 
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element of the set" were not analyzed, since the test 
instrument was designed to produce no errors for that 
concept — making it difficult to analyze a set error 
pattern. 
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V. CONCLUSIONS AND RECOMMENDATIONS 



The proposed complete response evaluation approach 
described here appears promising. Test results have dem- 
onstrated that the developed technique provides the 
educator with a method to structure the CAI course material 
towards responses which a student is more likely to pro- 
vide and, therefore, will prove to be most effective for 
the individual student. As it stands, the approach is 
ready for use by CAI lesson writers and evaluation per- 
sonnel. This final section summarizes the observed 
strengths and weaknesses of the proposed technique, 
suggests specific areas for further development, and out- 
lines a number of recommendations for utilizing the 
response evaluation approach. 

A. EFFECTIVENESS OF THE RESPONSE EVALUATION PROCESS AND 

SUGGESTIONS FOR FURTHER DEVELOPMENT 

On the basis of test results discussed in the preceding 
sections of this report, it can be concluded that the re- 
sponse evaluation process for a particular concept was 
reasonably effective; substantially more than the current 
processes that it may replace. Use of this approach re- 
sulted in the dramatic reduction of the infinite number of 
possible theoretical incorrect responses for a particular 
concept to a mere three. Also, substantive, although not 
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striking, improvements in the methodology for administering 
the final instrument were discovered. 

It must be admitted that, in terms of one of the more 
central measures of success, discovery of an response error 
pattern for each individual subject, is not as dramatic as 
might have been hoped for. A set error pattern was dis- 
covered for some individuals , but could not be generalized 
for all test subjects. 

Two major processes of the response evaluation approach, 
the methodology used, and the error response pattern will 
now be examined and discussed in some detail. 

1 . Methodology Used 

The imagery and verbal response methodology is sound. 
Although there is no proof that imagery dramatically reduces 
problem-solving errors (Weissinger-Baylon, 1978) , it is an 
effective method for evaluation of thought processes for the 
particular application for which it was used. It must be 
noted, however, that some problem-solving tasks make heavy 
demands on imagery, while others do not. This is not the 
result of the inability of the subjects to produce them, 
but on the inherent nature of the task. While problems 
were encountered in two areas , they were more of the nature 
of management problems rather than problems with the metho- 
dology per se. 

First, the test subjects were unaware of the process of 
using imagery in their problem-solving task and were re- 
luctant and not ready to accept its use. Subjects had to 
be continually urged to put down any mental images, if and 
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when they occurred. This reluctance to describe mental 
images is not uncommon in the imagery controversy , which 
dates back to the early part of this century. The imagery 
controversy raged between psychologists with Holt (1964) 
and Titchener (1926) pleading for a new and more direct 
consideration for mental imagery, while behaviorists re- 
jected the idea as unscientific. It was not until the mid- 
sixties when Shepard (1978) and others discussed the im- 
portance of imagery in the problem-solving role, that it 
was more readily accepted by cognitive psychologists. 
Therefore, in addition to the recognized newness of the use 
of imagery in problem-solving, the further reluctance of 
test subjects to use mental images to solve a problem- 
solving task may best be explained by Amheim: 

images must be highly abstract since the mind operates 
often at high levels of abstraction. But to get at 
these images is not easy. I mentioned that a good deal 
of imagery may occur below the level of consciousness 
and that even if conscious, such imagery may not be 
noticed readily by persons unaccustomed to the awkward 
business of self-observation. At best, mental images 
are hard to describe and easily disturbed. 

(Amheim, 1969, p. 116) 

There is no doubt that the use of mental imagery in 
problem-solving can be, and hopefully will be refined 
through subsequent psychological studies. Once the sub- 
jects are aware of the role of mental imagery, their 
reluctance to report it will diminish and its management 
will become more efficient. 
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The second area concerned over-emphasizing verbal re- 
porting versus the written reports. Since it is easier to 
speak than to write, some subjects initially tended to 
"overplay" their verbal responses, while "down-playing" the 
written protocol. Even though more accurate and complete, 
analysis and management of the taped verbal protocols 
became difficult when it could not be compared to the 
subject's written protocols. Further, to consider the 
opposite extreme, some subjects tended to shy away from 
any verbal responses when they discovered that they were 
to be tape recorded. Therefore, even though tape record- 
ings are more accurate and complete and, in some cases, 
also provide historical reference, their value is depend- 
ent on its application and its economic feasibility. 

2 . Error Response Pattern 

One of the main goals of this study was to discover a 
response error pattern for each subject. This discovery 
would have led to a more individualized package of instruc- 
tion for the intended user. However the results were less 
than dramatic, showing irregularities and inconsistencies 
in subject responses. For example, even though test re- 
sults for some subjects definitely indicated that they 
had learned an incorrect form of the concept, obtaining 
an error rate as high as four out of five possible errors, 
it was not conclusive. An important question that could 
be asked from these results: "if the incorrect response 

form was definitely learned, why were not five out of five 
errors produced?". Another example, showing irregularities 
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or "funny stuff" in subject's responses are shown in test 
results where the subject is successfully progressing 
through the problem, but then supplies the wrong response. 
The principal challenge, therefore, lies in analyzing the 
protocol data and answering the following questions: 

(1) How may these inconsistencies and irregulari- 
ties be explained? 

(2) Where do we make the cut-off mark to deter- 

mine if a subject has learned an incorrect 
response form: is it at fifty percent 

incorrect responses for a particular con- 
cept or dare we raise or lower this mark? 

These are difficult questions. A possible solution to 
the first question may best be explained by Bruner, 

Goodnow and Austin: 

. . . strategies as employed by people are not fixed 

things. They alter with the nature of the concept 
being sought, with the kinds of pressures that exists 
in the situation, with the consequences of behavior, 
etc. And this is of the essence. For what is most 
creative about concept-attainment behavior is that the 
patterning of decisions does indeed reflect the de- 
mands of the situations in which the person finds 
himself. 



(Bruner, Goodnow, Austin, 1956, p. 55j 
In addition, Feigenbaum (1963) pointed out that some- 
times the process we use to discover or reveal something, 
produces incorrect responses. The interesting question is 
"how often"? 

There have been many theories on how the human mind 
processes a problem-solving task. One of these, which 
seemed to be the most plausible, and from the protocol 
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data, the one generally followed in this study, stated that 
the problem-solving process depended upon the presence of 
certain previously learned rules (or in simpler cases, 
concepts) . These previously learned rules and concepts 
had to be recalled, as a first step, using instructions, or 
in the case of this experiment — the upper bound definition. 
Whatever stimuli was used to recall the previously learned 
rules, the externally applied guidance presumably functions 
to increase the vividness (or "availability") of some 
previously learned entity. 

At the next stage, the individual searched for and 
selected the recalled rules that were specifically relevant 
to the stimulus situation, and rejected those which were 
not. There is bound to be some "noise" at this stage of 
the game, and the process is one of distinguishing the 
noise from the relevant signals. For example, if the sub- 
ject is searching for the upper bound of a set, the 
geometric progression of the set may be quite irrelevant. 

If he is searching for the trigonometric relationship for 
"tangent" at different values for "pi," the size of the 
triangle may be quite irrelevant. Conceptualization of 
these features of the stimulus situation may have been re- 
called; but they must be distinguished from other concepts 
and discarded. 

Combining subordinate rules was the next stage of the 
problem solving process . There are many ways in which 
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recalled rules and concepts may be combined; the object is 
to find the correct one. For example, consider set five: 

Set 5: {1, 1/2, 1/3,...} 

The three dots (...) would indicate the rule that the set 
is infinite. However, the set also stresses consideration 
for the rule when the numbers are progressing in descending 
order. Presumably, in this stage, the rules were narrowed 
to a relatively few likely combinations , and thus reduced 
the time of search. 

At the next stage, the subject arrived at a provisional 
rule which he believed may solve the problem. This rule 
was then subjected to verification by carrying out the 
operations it suggested with reference to the stimulus 
objects . 

The final stage that the subject was found to perform 
was verification. Here, the provisional rules were 
checked by application to the specific problem. Should 
verification not work, the subject returned to the task of 
trying new combinations of rules. When verification did 
work, the solution had been acquired. 

In summary, the problem solving process used may be 
thought of as a linear sequence of operations that are 
actively carried out by the subjects according to a system 
of links influenced by a set of rules and laws. The in- 
consistencies and irregularities of the subject's responses 
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may be a result of the breaking of these links or wrong 
applications of the rules. Mackworth (1965) compares the 
human mind to a very efficient computer, composed of a 
large number of programs to handle all required situations 
(with modifications) . He states that the principal 
criteria of problem solving is "choosing correctly between 
existing programs." Incorrect selection of these programs 
will provide incorrect or inconsistent solutions to the 
problem-solving task. 

The answer to the second question must be made on a 
judgemental decision, taking into consideration the 
objectives of the particular concept. The analysis has 
indicated, however, that if there was less than fifty per- 
cent error response rate for a particular response error 
form, a definite error pattern could not be determined. 

In conclusion, the many unanswered questions for de- 
fining a specific error response pattern must be left for 
the behavioral psychologists. In this study, the attempt 
was made to discover a specific error pattern for a par- 
ticular concept, using mental imagery as a tool. A few 
explanations were provided for the inconsistencies and 
irregularities for subject's responses, but much remains to 
be done in this area of response evaluation. 
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B. RECOMMENDATIONS FOR THE RESPONSE EVALUATION APPROACH 

IN THE CAI ENVIRONMENT 

Both Abrahms (1971) and a report on "materials develop- 
ment procedures and evaluation," by the McDonnell Douglas 
Astronautics Company (1980) , indicated that a major short- 
coming in CAI development is in the area of response 
evaluation. They stressed a need for a method which would 
guide the lesson writer, by defining more sophisticated 
rules by which anticipated responses to constructed re- 
sponses may be judged. The technique provided in this 
study seems to fulfill some, if not all, of these require- 
ments. 

The major conclusions drawn from this study pertain to 
the finding that the response evaluation approach can be 
used to develop CAI lessons with the individual student in 
mind rather than to provide an additional step in the 
already tedious writing of course materials per se. Thus, 
the intent of the recommendations outlined below is to 
suggest potential benefits to be derived from broader 
application of an instructional technology which has been 
shown to be reasonably effective in this context. 

An obvious next step would involve further development 
and implementation of the response evaluation approach in 
the actual CAI course material development process . The 
area which appears particularly promising is the identifi- 
cation of the principal incorrect responses for a particular 
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concept. The educator may now define his lesson branching 
structure with these incorrect response forms in mind. 
Providing such system responses as : 

(1) "Are you aware that plus infinity is not the 
upper bound for all sets," 

(2) "Are you aware that an infinite set may have 
an upper bound," 

(3) and finally, "Are you aware that the upper 
bound for a set does not necessarily have to 
be a member of the set." 

With some modification and further study, the response 
error pattern process could be implemented to provide for a 
more totally structured individualized CAI system for a 
particular concept. 

Eventually, one of the most promising areas for the 
response evaluation approach would be to broaden the area 
of CAI instruction to a more diversified field of 
knowledge. For example, CAI lessons could be prepared for 
fields which were generally considered too complex due to 
their psychological response processes for their mastery. 

Finally, it is recommended that any implementation of 
the response evaluation process be coupled with the follow- 
ing guidelines, proposed by Colonel Joe Nastasi, if the 
CAI lesson is to be a success: 

(1) CAI lesson writers must be senior instructors 
who are experienced in their subject matter 
areas and in the course to be developed. 
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(2) CAI lesson writers should be trained programmed 
instruction writers. 

(3) Learning how to write viable and valid lessons 
takes considerable time, and constant turnover 
of authors is disastrous to satisfactory 
progress . 

(4) CAI lesson writers must be assigned full time 
duty of producing lessons for the CAI program. 

(5) Close and continuous liaison must be main- 
tained between the course, other instructors, 
CAI lesson writers , the educational con- 
sultant and the CAI project operations. 

(6) An authorized course subject block diagram, 
with indicated lessons, examination questions, 
behavioral objectives, plan of each lesson, 
and lesson flowcharts must be prepared prior 
to writing and coding lessons , and be avail- 
able to insure valid CAI lesson development. 

(7) The officer in charge and chief instructor of 
the course being developed must retain full 
control and supervision of the CAI lesson 
writers and their efforts. 

(8) All CAI lessons must be very carefully re- 
viewed from a pedagogical and student view- 
point. 

[Prokop, 1976, p. 218-219] 
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APPENDIX A 



TEST DATA FROM SUBJECTS 



SI C.D. 



SI C.D. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 

m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 

1. E£ R 

2. m£ R 

3 . V x — m m = upper bound . 



PRACTICE PROBLEM : Let E be the set £l, 2, 3, 4j , i.e. the set consisting 

of- the first four positive integers. Does E have an upper bound? 



1. See four finite elements. 

2. Looked for greatest value. 

3. Yes, has upper bound of 4. 

SET 2: The set of natural numbers, N (i.e. the positive integers) 

1. I see an infinite set. 

-| ? 

2. An infinite set is unbounded. 

3. No. 

SET 4; A finite set. 

1. No mental image at first. Thought about it. 

Then saw 100 tanks. 

1 ’2 3 100 

0 0 t • « • • 
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Si C.D. 



2. Yes, there is upper bound of greatest element in set. 
SET 5 : 1, 1/2, 1/3, ... 

1. I see infinitely smaller numbers. 




2. Therefore upper bound must be the greatest number in the set. 
SET 6 : The set of real numbers, R. 

1. No. 

2 . An infinite set ( r£)unbounded) . 

SET 11 : The non-negative numbers. 

1. Infinite set starting at 0. 

^ 

2. No, same as set 6. 

SET 13 ; The closed interval |_3, 5^. 

1. Yes. 

2. The largest element in set is upper bound. 

3 . Finite set . 

SET 17: & vj, v ir, xr, 'Sf, . . . ^ 

1. Yes. 
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Si C.D. 



2. after test 
No. 

SET 18 : ^Tan x, where x -Tf, 2lf, 3 rf, 5lf, . .. J 

1. Yes. (guess since tangent is bounded by 0 , 1) . 

2. I see an set of 0's and l's 

I i D i 1 i • • - | 

SET 21 ; £l, 2, 4, 8, 16, ... j 

1. I see numbers increasing. 

2. No, infinite set. 
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S2 L.M. 



S2 L.M. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 



1. Definition thought: 



The real numbers ordered on a number line. 



-3-2-10123 
4— I 1 1 1 1 1 1 — > 



PRACTICE PROBLEM : Let E be the set £l, 2, 3, 4^, i.e. the set consisting 

of the first positive integers. Does E have an upper bound? 



1. 12 3 4 

H — I 1 K? 

2. upper bound = 4. 

SET 2 ; The set of natural numbers, N (i.e. the positive integers) 
1. 1 2 3 ... 



2. No bound, it goes to infinity. 
SET 4: A finite set. 



1. Points on the line — bounded by the element which falls farthest 
to the right on the number line (like definition) . 

SET 5 : £l, 1/2, 1/3, ...} 



1. I see an infinite number of points getting smaller. 
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S2 L.M. 



1/3 1/2 

3. bounded by 0. 

4. I remember definition and it asks for upper bound, therefore 1. 

SET 6 : The set of real numbers, R. 

1. no bound. 

2. v> 

SET 11 ; The non-negative numbers. 

1. I see an infinite set. 

2. Upper bound >0^ (the first non-negative number). 

3. ... -2 -1 0 

< 1 1 1 

4. Qfo lower bound J 

SET 13 : The closed interval C? * 5 j . 

1. Saw the largest number 5. 

2.3 5 

-E 1 3- 

3 . bound 5 . 

SET 17 : £V5", V3, V4, V5, vg, V=7, ...J- 

1. No bound — each number bigger than the previous number. 
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S2 L.M 



SET 18 : £ Tan x, where x = 7f ,2 4^, 5 77) 67T, . ..^. 

1. I could not remember definition of tan. 

2. I see ... , therefore goes to infinity. 

3. No bound. 



SET 21 : £l, 2, 4, 8, 16, ... J . 
1. Increasing infinite set. 



1 



12 4 8 



2. no bound. 
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S3 T.V. 



S3 T.V. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 

1. xi m for every x in E. 

PRACTICE PROBLEM : Let E be the set£l, 2, 3, 4^, i.e. the set consisting 
of the first four positive integers. Does E have an upper bound? 

1. Yes, 4. 

2. Four is the largest number and so all of the others are less than 

or equal to it so it must be the upper bound. 

SET 2 : The set of natural numbers, N (i.e. the positive integers). 

1. No upper bound sp infinite. 

2. Imagery — (1, 2, 3, ...} 

SET 4 ; A finite set. 

1. I see a certain number of points. 



2 . Imagery 




3. Upper bound equals largest number in set. 




1. 1 is upper bound. 



2. I visualize the numbers trailing off to right so 1 is largest. 
My first impression was that it was an infinite set with no 
upper bound. 
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S3 T.V. 



SET 6 : The set of real numbers, R. 

1. Infinite set, no upper bound. 

2 . Imagery 1 , 2 , 3, 4, . . . ^ , 

SET 11 : The non-negative numbers. 

1. Infinite set. 

2. no upper bound (see above). 

SET 13 : The closed interval [3- 3], 

1. 5 is the upper bound. 

2. It is the largest number in a finite set. 

SET 17 ; £V2,V^,N/4,V§,V6,V7 / ... J 

1. Infinite set — no upper bound. 

2. I first look at the values of the perfect squares. 

3 . Imagery 

I 1 > 9 

12 3 



SET 18: 



^Tan x, where x = 7T t 27, 37 t, $7 f 57, 67, 




1. Visualize coordinate plot of trig values. 

2. tan = sin 

cos 
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S3 T.V 



3. Infinite set — no upper bounds. 



SET 21: 



£ 1, 2, 4, 8, 16, ...J 



1. Infinite set no upper bound. 

2 

2. Imagery = X 
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S4 R.W. 



S4 R.W. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 

1. DEF: For all answers oo is no_ bound 

PRACTICE PROBLEM : Let E be the set £ 1, 2, 3, 4^, i.e. the set consisting 
of the first four positive integers. Does E have an upper bound? 



1. E has an upper bound. 

2. m = 4, because set E has ascending order. 

3. 12 3 4 

| i 1 I 

SET 2 : The set of natural numbers, N Ci.e. the positive integers). 

1. 1 2 3 4 ... +oO 

» 1 1 1 — 

2. There is no upper bound, unlimited. 

SET 4: A finite set. 



1. A finite set not necessarily has an upper bound, because "finite" 

is concerned with the number of elements in the set. 

2. After asking examiner — this set is bounded above. 

SET 5 : £ 1, 1/2, 1/3, ... j 

1 . 1 0 

l w — *-* — I — 

2. The upper bound of this set is 1, because the set has a descending 
order in its elements. 
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S4 R.W. 



3. m = 1 

SET 6 : The set of real numbers , R. 

1. OO — 0 + oo 

1 

2. The set of real numbers has no upper bound because they go to + 

SET 11 : The non-negative numbers. 

1 . t . 

0 

2. No upper limit, goes to oO 

SET 13: The closed interval &. s]. 

1. [3, 

2. The closed interval has an upper bound. 

3. m = 5 . 

SETJ^: £V2,\/3,V / !4,'^5,V6,V7, ...J 

1. A v5 v£ ... cO 

^ 1 » i 

2. no upper bound 

SET 18 : £ Tan x, where x -IT , 2 T, 3v7 5^ 6 if, . ..J' 

1 . 




S4 R.W 



2 . 

3. 



Set contains all zeroes. 
Set has no upper bound. 



E^O, 



a. 



a. 




SET 21 : { 1, 2, 4, 8, 16, ...J 



1. Set has no upper bound. 

2. Set of the squares of natural numbers 



-> < 2 ° 
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S5 S.L. 

DEFINITION : Let E be a subset of the real numbers, R. An element, m 
of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 




3. x =[e]'— m 

4. m = element in E. 

PRACTICE PROBLEM : Let E be the set {l, 2, 3, 4}, i.e. the set consist- 

ing of the first four positive integers. Does E have an upper bound? 



1. [Ej = (1, 2, 3, 4) 

2. m = 4 

SET 2 : The set of natural numbers, N (i.e. the positive integers). 

1. Natural numbers, N, 5 unlimited positive number. 

2. So there should not be any upper bound. 

SET 4 : A finite set. 

1. A finite set — .> the set is finite. 

2 . So should be an upper bound of the subset E. 

SET 5: {1, 1/2, 1/3, ... } 
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1. Cl, 1/2, 1/3, ...1 unlimited. 

2 . Fraction 1/n . 

3. 1 = the upper bound. 

SET 6 : The set of real numbers, R. 

1. Undefined property 5 - 

2. The set of real numbers, R,_ 5 >if it means all real numbers _^then 

there is no upper bound. 

3. If it starts from a certain number downward ^ there is an upper 

limit. 

SET 11 : The non-negative numbers. 

1. All non-negative numbers . 

2. No upper bound except 

SET 13 : The closed interval Qi , 5j. 

1 . » i 

3 5 

2. 5 = upper bound. 

SET 17 : ^V / 2 /V '3,n/4,\/5,V6,v'7, ...J 

1 . (VI ;S3 , ... , ... } ==> Vn 

2. no upper bound. 

SET 18: C Tan x, where x -IT, 2 if, 3 7f, 4 ^ 5 7f, &7f, ... 
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Tan T? = Tan 2 1? 



2 . upper bound = 0 



SET 21 



: £ 1, 2, 4, 8, 16, ... J 



1. See an infinite increasing set. 

2. no upper bound 2 m , 
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S6 P.H. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 

1. mC R. 

PRACTICE PROBLEM : Let E be the set£l, 2, 3, 4_} , i.e. the set consisting 
of the first four positive integers. Does E have an upper bound? 



1. Yes 

2. upper bound — 4. 

SET 2 : The set of natural numbers, N Ci.e. the positive integers). 

1. N = £l, 2, 3, ...J 

2 . Yes , always a real number that greater or equal to the largest 

integer. 

SET 4 ; A finite set. 

1. A =£al, a2, ..., ai^ 

where i is finite number 

2. yes bounded. 

SET 5 : £ 1, 1/2, 1/3, ...J 

1. The set decreases. 

2. Yes, bounded = 1. 
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SET 6 ; The set of real numbers, R. 

1. <r -> 

2. No, real numbers not bounded upper side. 

SET 11 : The non-negative numbers. 

1. No 

2. Non-negative numbers could be real — $> so the same as set 6 above. 

SET 13 : The closed interval | 3, 5^ 

1. 3 5 

I * 

2. Yes, bound Z 5. 

SET 17 : ^^2,'/3 / V3 > V5,V6 f V7 / ...J 

1. Yes, because the number under square root are integers; we always 

could find a real number that is: larger than that. 

2. \/3 V4 ... VH 

! / 1 1 

SET 18 : ^ Tan x, where x -V , 2 IT 1 , 2% 4^, 5^ 6 V, . . . J» 

* # 

2. always tan x = 0 

3. yes, an upper bound 
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SET 21: 




4, 8, 16, 




1. Set of integers. 

2. Yes would be upper bound. 

3. Same as set 2. 
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S7 Y.K. 

DEFINITION : Let E be a subset of the real number, R. An element, 

m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 

PRACTICE PROBLEM : Let E be the set^l, 2, 3, 4 J, i.e. the set consisting 

of the first four positive integers. Does E have an upper bound? 

1. Yes, the upper bound is 4. 

2. Each number in the set is less than 4. 

SET 2 : The set of natural numbers, N (i.e. the positive integers). 

1. Yes 

2. There is a number greater among the other numbers. 

3. Examiner questions answer: 

4. Infinity is the bound. 

SET 4 : A finite set. 

1. Yes, same as set 2. 

SET 5: £l, 1/2, 1/3, . . . j 

1. yes 

2. The upper bound is 1 

3. Each number in the set is less than 1. 

SET 6 : The set of real numbers, R. 

1. Yes, same as set 2. 
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SET 


11s 


The non-negative numbers, 


i. 


Yes, 


same as set 2. 


SET 


13: 


The closed interval jj3, l 


i. 


Yes. 




2. 


5 is 


the upper bound. 


SET 


17: 


^\/2 ,\/3 ,\/4 ,V5 ,V6 ,\/7 , . . . J 


1. 


no, : 


because it is infinite. 


SET 


18:^ 


■ Tan x, where x =77, 2/7^ 


1. 


No upper bound 


2. 


Goes 


to infinity 


SET 


2!: j 


f 1 / 2, 4 , 8, 16 , • • 




1. No because it is infinite. 
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S8 R.F. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 

i- i *} E 

2 . ^ ... m ^ 

subset E of real numbers, R. 

3. x m 

4. m is only an upper bound of subset E if x - m 



© 



PRACTICE PROBLEM : Let E be the set \ 1, 2, 3, 4J- , i.e. the set consisting 

of the first four positive integers. Does E have an upper bound? 



1. E has an upper bound (.41 

2. Reason is that it is a finite set, so the highest number is the upper 

bound. 

3. E = £l, 2, 3, 4^ 

SET 2 ; The set of natural numbers, N (i.e. the positive integers \ 

1 *^ 1 , 2 1 3 , • . • t 

2 . infinite set 

3. no upper bound 
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SET 4 ; A finite set 

1. £l, 2, 3, 4, 5 J 

2. has upper bound — in my ex. upper bound is 5. 

SET 5 : £ 1, 1/2, 1/3, . . . J 

1. decreasing set to infinity. 

2. def. of upper bound means the highest number in that set. 

3. The upper bound, in ex., is the highest number Cll. 

SET 6 : The set of real numbers, R. 

1. Does not specify what R contains. 

2. It is an infinite set. 

3 . Correction — it does specify what R contains — all real numbers . 

4. All real numbers is an infinite set. 

5. Thus no upper bound. 

SET 11 ; The non-negative numbers. 

1. Means all positive numbers. 

2. All positive numbers is an infinite set. 

3. No upper bound. 

SET 13: The closed interval L 3 ' 5 J 

1. Contains only numbers 3 and 5. 
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2. Finite set 

3. Upper bound is 5. 

SET 17 : £V2 //3 ,V4 ,Vs ,V6 ,\/7 , . . . J 

1. Increasing set. 

2. Increasing by one square root digit. 

3. In order, so that the next number will be 

Va, V9, Vio, ... ,Vn J 

4. It is an infinite set. 

5. Does not have an upper bound, except for the Vn , which is undefined 

at present. 

6. I would say this set does not have an upper bound. 

SET 18 : £ Tan x, where x = /T, 2^ 3TT, 47T, 5/7, 6*7, . . 

1. Increasing set. 

2. Same as problem set 17. 

3. No upper bound. 

SET 21 : £ 1 , 2, 4 , 8, 16, ...j- 

1. increasing set in sequence of 2 A 

2 . infinite set . 

3. No upper bound. 
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S9 F.K. 

DEFINITION ; Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 




© 



PRACTICE PROBLEM : Let E be the set£l, 2, 3, 4,J , i.e. the set consisting 

of the first four positive integers. Does E have an upper bound? 



1. I see a finite set. 

2. 12 3 4 

» . . * • 

3. According to definition xi.fi m 

4. Yes, upper bound = 4 or greater. 

SET 2 ; The set of natural numbers, N (i.e. the positive integers).. 

1. what is a natural number — is 0 a natural number? 

2. This set goes to infinity 

? «*> 

* — i 1 

0 1 2 ... 
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3. Since infinity is the largest number — the upper bound is 
infinity. 



SET 4: A finite set. 




2. This is just like the practice prohlem. 



3 . Upper bound . 



SET 5 ; £l, 1/2, 1/3, ...J 



1. I see the set decreasing. 

1/4, 1/5, 1/6 => 1/n 

3. Since it is decreasing the largest number must be the upper bound. 

4. Yes, upper bound = 1. 

SET 6 ; The set of real numbers, R. 

1. what is a real number? 

2. Rational numbers such as -3/4, 18/29-, 3, -6 



4. It (real) goes to infinity. 

5. Infinity must be the upper bound, since there is no number greater 

than infinity. 

SET 11 : The non-negative numbers. 

1 . I see a line going towards the left to - cP 



3. Irrational numbers such- as N/2 





-a - > 



o 
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2. Therefore zero must be the upper bound. 

3. Wait, it wants non-negative numbers. 

4. The line goes to the right to + oo 

5. Since + is the largest possible number, it must be the upper 

bound. 

SET 13 : The closed interval Jj3, 5j 

1. I see a line like this 

b d 

2. From previous experience this is a finite set, with boundaries 

3 and 5 

3. upper bound of 5. 

SET 17 : ^V2 ,V5 ,V3 ,V5 ,V% , . . . J 

1. I first see an infinite set. 



2. I then look at the "V - ”” 

3. I ignore this, since infinity suggests an upper bound. 

4. Yes, upper bounded above. 



SET 



18 : £ Tan x, wherex = Tf , 2% zK 5 If, <oT, . . . J 



1. I see an infinite set. 

2. I immediately say, yes bounded above. 

3. I look at what tangent means 



sin 

tan 

cos 
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4. what are the values for tan at 7f , 2 Tf , etc 

5. I draw sine curve and cosine curve. 



sa F.K 



| OoQc> ( 

6. Determined that tan x = ^Q,0,0,... 

7. Upper bound of 0 or greater. 



SET 21: 




2 , 



4, 



8 , 



16, 



•} 



1. I see a progression — 2 X 

2. It goes on — no upper bound. 
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SIO D.M. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 

m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 




2 . x ^ m ^=^R 

PRACTICE PROBLEM : Let E be the set fl, 2, 3, 4 } , i.e. the set consisting 

of the first four positive integers. Does E have an upper bound? 



1. This problem is obvious. 

2 . 1234 

* > 1 » 

3. it is finite, therefore upper bound of 4. 



SET 2 : The set of natural numbers, N Ci.e. the positive integers). 

1. I immediately see an infinite set going towards plus infinity. 

•f oO 

* > 

2. This implies to me that no upper bound exists. 

SET 4: A finite set. 



1. Obvious, no upper bound. 
SET 5: £l, 1/2, 1/3, ... j. 
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1. I see an infinite set. 

2. It decreases to zero —^therefore , the largest number must he the 

upper bound. 

3. Almost tricked me by showing infinite set. 

SET 6: The set of real numbers, R. 



1. I immediately see an infinite set, which causes me to say no upper 

bound. 

2. I see no other images. 



SET 11 : The non-negative numbers. 

1. I draw a picture 




0 



2. Since 0 is the first non-negative number, it must be. the upper bound. 
SET 13 : The closed interval f3, sj 

1. There are an infinite number of points between 3 and 5, but endpoints 
are 3 and 5 . 



3 5 

t i 

2. Yes, upper bound. 

SET 17 ; £ S2 ,\/3 , V4 ,V3 ,n/6 , ...J 

1. I see an infinite set. 

2. I ignore the " V " 
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3. Set goes to infinity 

4. No upper bound 

SET 18 : £ Tan x, where x =7f , 2 Tf, zff, 4/^ 5/^ 6/^ ••*3’ 
1. Tan x = x/y 




3. whenever x = 0, y is either 1 or -1 

4. therefore tan x =|o, 0., 0, . . 

5. Set has an upper bound 

SET 21 . £ 1, 2, 4, 8, 16, 

1. I see an infinite set. 

2. Progression = 2 n 
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Sll s.w. 

DEFINITION : Let E be a subset of the real numbers, R. An element, 
m, of R is said to be an upper bound of E if x is equal to or less than 
m for every x in E. 



1. x-^E_^* 



2 . 




3. m R 

4. x ^ m 

PRACTICE PROBLEM : Let E be the set£l, 2, 3, 4 3 t i.e. the set consisting 

of the first four positive integers. Does E have an upper bound? 



1 . 




2. Since E is a finite set, there exists a R greater than E. 

3 . Upper bound . 



SET 2 : The set of natural numbers, N Ci.e. the positive integers!. 

1. No real images. 

2. Since natural numbers go on indefinitily , it implies an infinite 

set. 

3. No upper bound. 
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SET 4 ; A finite set. 

1. How can I assign an m? 

2. I don't see any numbers or elements. 

SET 5 ; £l, 1/2, 1/3, ...J 

1. I see 

• • • 

2. Infinite set has no bound, but wait it is decreasing. 

3. I draw picture 

<— < * — 1 

1/3 1/2 1 

4. The progression is 1/n _ ^ goes to 0 

5. Upper bound is the largest number — 1. 

SET 6 : The set of real numbers, R. 

1. I see this immediately as an infinite set. 

Therefore not bounded. 

2. No mental pictures appeared. 

SET 11 ; The non-negative numbers. 

1. Draw picture of the non-negative numbers. 

* > 

0 + 

2. Goes to infinity. 

3. No upper bound. 
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SET 13: The closed interval 



C 3 ' 5 ] 



1. First looked at bracketed numbers and recognized m=5. 

2. Next looked at the word interval this implied infinite set. 

3. Since it is bracketed 5* 



SET 17 :^S /2,y/3,sfiyi yiyj, ... } 



1. Saw numbers with square root signs. 

2 . Saw- dots which implied infinity and no way of assigning an m 

3. numbers are increasing under square root signs. 

4. infinite set, m can not be defined. 



SET 



18 : ^ Tan x, where x -If , 2 TT, 7Jf, 47^, 5^ 6 Tf, . . . ^ 



1 . 

2 . 

3. 



First saw an infinite set. 

The value If for tan looks familiar. 
Draw picture 




4. 

5. 

6 . 



tan x = sin x 
cos x 



IT = 0 2 TT = o 



the set is o, o, * * * * 

upper bound. 



SET 21; 



£ 1, 2, 4, 8, 16, ... ^ 



1. infinite set came to mind first. 



107 



Sll s.w 



2 . Same as set 17 . 

i 

, 

! 
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